Reducing synchronization of tasks in latency-tolerant task-parallel systems

ABSTRACT

Techniques are provided for reducing synchronization of tasks in a task scheduling system. A task queue includes multiple tasks, some of which require an I/O operation while other tasks require data stored locally in memory. A single thread is assigned to process tasks in the task queue. The thread determines if a task at the head of the task queue requires an I/O operation. If so, then the thread generates an I/O request, submits the I/O request, and places the task at (or toward) the end of the task queue. When the task reaches the head of the task queue again, the thread determines if data requested by the I/O request is available yet. If so, then the thread processes the request. Otherwise, the thread places the task at (or toward) the end of the task queue again.

FIELD OF THE DISCLOSURE

The present disclose generally relates to task scheduling and, moreparticularly, to reducing synchronization of tasks.

BACKGROUND

In computing, a task is a unit of work. A process or thread is said to“execute” a task if the process or thread performs the unit of work orcauses the unit of work to be performed. Because a program may requiremillions of tasks to be executed, efficiently scheduling the tasksbecomes extremely important. If tasks are not scheduled efficiently,then the program may take a significant time to execute.

One approach for implementing a task scheduling system is to providecalls to put a certain task to sleep and wake it up again. Tasks may bemultiplexed over worker threads, with each worker thread executing tasksfrom a ready queue. If a task becomes blocked by, for example, issuingan I/O operation, then the task is “put to sleep” by placing the task ina wait queue, which may be one of many wait queues, each for a differenttype of I/O operation. Once the I/O operation completes, the task iswoken up using the wake operation. “Waking” a task involves dequeuingthe task from the wait queue and enqueuing the task in a ready queue sowork for the task will be resumed once the task reaches the front of theready queue.

Under this approach, the ready queue becomes a shared resource betweenthe task executing thread and the thread handling callbacks generatedupon completion of I/O operations. Because multiple threads access theready queue and the wait queue, synchronization constructs are added toeach queue operation, which results in adding overhead to taskoperations. Such overhead may not be significant if the overhead issmall relative to latency of the I/O operations.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a block diagram that depicts an example system of networknodes, in an embodiment;

FIG. 2 is a block diagram that depicts a network node, in an embodiment;

FIG. 3 is a flow diagram that depicts a process for processing tasks, inan embodiment;

FIG. 4 is a block diagram that illustrates a computer system upon whichan embodiment may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

New kinds of applications require frequent communication among nodes ina network but a relatively low computation-to-I/O (communication) ratio.An example of such an application is distributed graph analysis. Suchapplications typically require many small computations where each singlecomputation requires many remote data accesses. Fortunately, theseapplications also exhibit a great degree of parallelism. Therefore, longcommunication latency can be hidden through context-switching betweenlarge numbers of concurrent computations between which each computationdoes not depend on other computations.

However, in such cases, the aforementioned task queue overheads can besignificant in performance. Due to a large number of concurrentcomputations and small computation-to-communication ratio, taskswitching (i.e., where a task moves from a ready queue to a wait queueand back to the ready queue) happens very frequently and, therefore,overheads in task switching accumulate rapidly and become significant.

According to techniques described herein, a task queue does not become ashared resource. Instead, a task executing thread maintains soleownership of a task queue. Such an arrangement is advantageous becausethere is no need for synchronization between multiple threads. Removingthis synchronization can lead to improved performance and scalability.Such an arrangement is beneficial in applications where the time takenfor an individual I/O operation to complete is significantly less thanthe time taken for a thread to execute all of the other tasks that arein the task queue. By the time the thread has finished the other workavailable to the thread in the task queue, the I/O operation that thethread has issued is likely to have completed.

Once a task is inserted into a task queue, the task (even if it requiresan I/O operation) is not reassigned to other queues. Instead, an I/Otask is reinserted into the task queue while the I/O task “waits” for anI/O operation to complete. Avoiding the use of wait queues isadvantageous because there is no need for synchronization between athread making a task wait and a thread waking a task. Furthermore,avoiding the need to transfer a task from a wait queue to a task queuereduces the latency in waking a task

In an embodiment, no more than a single working thread in a computingdevice is allowed to process tasks in a task queue. In a relatedembodiment, multiple threads can access tasks in a task queue. Forexample, initially, each of multiple threads is assigned its own taskqueue. Later, a first thread processes a task from a task queue assignedto a second thread if there are no more tasks in the task queue assignedto the first thread. Task queue sharing (by multiple threads) may beefficient if synchronization between threads running on the sameprocessing core is efficient.

Although the following description includes examples related to graphanalysis, embodiments are not limited to any particular type of dataprocessing.

System Overview

FIG. 1 is a block diagram that depicts an example system 100, in anembodiment. System 100 includes three network nodes 110, 120, and 130.Although three network nodes are depicted, embodiments are not limitedto a system of three network nodes. Instead, embodiments are applicableto systems with one, two, or more network nodes. For example, a task maywait on locks, barriers, or other synchronization operations with otherthreads in the same computing node or on I/O operations on a disk orother storage within the same computing node.

Each network node in system 100 includes one or more processors andmemory for storing instructions that are executable by the one or moreprocessors for performing techniques described herein. The memory ofeach network node stores results of operations and data that isretrieved from one or more other network nodes. Each network node mayalso include (or be connected to) non-volatile storage that stores datarequired by the network node.

Each network node in system 100 is communicatively coupled to each othernetwork node by a link. For example, network node 110 is connected tonetwork node 130 by link 142 and network node 110 is connected tonetwork node 120 by link 144. Links 142-146 allow network nodes 110-130to be directly connected to each or at least communicatively coupled toeach other. For example, one or more of links 142-146 may be a networkthat is implemented by any medium or mechanism that provides for theexchange of data between respective network nodes. Examples of such anetwork include, without limitation, a cluster interconnect network,Local Area Network (LAN), Wide Area Network (WAN), Ethernet or theInternet, or one or more terrestrial, satellite or wireless links.

Each network node in system 100 is configured to send requests for data,over one of links 142-146, to one or more of the other network nodes.For example, the requests may be for graph data, a different portion ofwhich is stored in different ones of network nodes 110-130. As aspecific example, if network node 110 is determining whether there areany graph vertices in common between the neighbors of vertex A and theneighbors of vertex B and network node 110 does not have data thatindicates the neighbors of vertex B, then network node 110 sends arequest for the neighbors of vertex B to network node 120 and/or networknode 130. If network node 110 does not know ahead of time which networknode stores (or has access to) the neighbors of vertex B, then networknode 110 may send a request for the data to both network node 120 andnetwork node 130.

In an embodiment, each network node is configured to know to which othernetwork node to send network I/O requests. For example, a graph may bedivided among network nodes 110-130 and each network node stores mappingdata that indicates which network node stores which neighbor data. Forexample, for a graph that includes vertices 1-3,000,000, mapping dataindicates that network node 110 stores neighbor data that identifiesneighbors for vertices 1-1,000,000, network node 120 stores neighbordata that identifies neighbors for vertices 1,000,000-2,000,000, andnetwork node 130 stores neighbor data that identifies neighbors forvertices 2,000,000-3,000,000.

Computing Node

FIG. 2 is a block diagram that depicts a computing node 200, in anembodiment. Computing node 200 may be any one of network nodes 110-130in FIG. 1. Computing node 200 includes a worker thread 212, a task queue214, an I/O interface 216, and storage 218.

While only one worker thread 212 is depicted in computing node 200,computing node 200 may include multiple worker threads, each of which isassigned to a different task queue of multiple task queues (notdepicted). The number of (software) worker threads may be dependent onthe number of hardware threads or cores in computing node 200. Forexample, if computing node 200 has a dual core processor that supportstwo hardware threads per core, then computing node 200 may have fourworker threads. Thus, computing node 200 may have four task queues, onefor each software thread. As another example, if computing node haseight quad-core processors with two hardware threads per core, thencomputing node may have 64 worker threads. Thus, the determination ofthe number of task queues to create and the number of worker threads toinitiate may be based on determining a number of processors that acomputing device (that will include the tasks queues and worker threads)includes.

Task queue 214 includes zero or more tasks. Task queue 214 has a “front”or “head” and an “end” or “tail.” Tasks that are at the head of taskqueue 214 are processed before tasks that are at the tail of task queue214. At any one time, task queue 214 may include thousands or millionsof tasks. Each task corresponds to one or more operations. When a taskis processed by worker thread 212, worker thread 212 causes an operationassociated with the task to be performed. If a task is associated withonly one operation, then, once worker thread 212 processes the task, thetask is removed from task queue 214. If a task is associated withmultiple operations, then worker thread 212 may cause all or some of themultiple operations to be performed. If not all operations areperformed, then worker thread 212 places the task at the end of taskqueue 214 (or at another position in task queue 214).

In an embodiment, task queue 214 comprises a set of pointers to tasks.In this way, the task themselves (which may be simple data structures ofone or more variables) do not need to be copied or moved around inmemory in order to place the tasks in task queue 214. Instead, memorypointers, which may be much smaller in size than their correspondingtasks, are added to and removed from task queue 214. Thus, as describedherein, “adding” a task to task queue 214 may involve storing the actualtask at a certain location in (e.g., the end of) task queue 214 oradding a pointer (that points to or references the task) to task queue214.

A task may be in one of multiple states. For example, the first timeworker thread 212 processes a particular task, the particular task isinitially in state A and advances to state B. The second and third timesworker thread 212 processes the particular task, the particular task isin state B and remains in state B. The fourth time worker thread 212processes the particular task, the particular task advances to state C.

Task queue 214 may be implemented in any number of ways, such as anarray, a vector, or a linked list. While a linked list may be moreflexible in allowing a task to be placed in any part of the queue, anarray and a vector are more space efficient than a linked list and canfacilitate faster operations on their head and tail.

Task queue 214 may be implemented as a FIFO queue. In that way, tasksare processed in the order in which the tasks are added to task queue214.

Worker thread 212 “consumes” or processes tasks in task queue 214.Worker thread 212 is part of an application program that, when executed,causes the tasks to be generated. An example of such an applicationprogram is a graph analysis application program that determines commonneighbors among nodes in a graph. In addition to consuming tasks in taskqueue 214, worker thread 212 may also be responsible for generatingtasks that are placed in task queue 214.

Alternatively, in an embodiment, a separate thread (not depicted) isresponsible for generating tasks and inserting them in task queue 214while worker thread 212 consumes tasks from task queue 214. Each threadhas access to data that indicates (a) a location of the head of taskqueue 214 (e.g., via an index value in an array) and/or (b) a locationof the tail of task queue 214. In this way, generation and consumptionof tasks may be performed concurrently. For example, a generating taskmaintains end location data that indicates a location in memory of theend of task queue 214 since only the generating task may need to knowthat location, while a consuming task maintains a start location datathat indicates a location in memory of the start of the task queue 214since only the consuming task may need to know that location.

Whether one thread or two threads may access task queue 214, thegeneration and consumption of tasks may occur in distinct phases thatmay or may not overlap. For example, in a first “generation” phase, oneor more threads may be responsible for creating tasks and inserting thetasks into one or more task queues. Then, in a second “consumption”phase, a different thread is assigned to each task queue and processesthe respective tasks. The generation phase may be required to completebefore the consumption phase begins. Alternatively, the consumptionphase may proceed as soon as a task is inserted into a task queue orafter a certain number of tasks (e.g., 1,000) are inserted into the taskqueue.

Computing node 200 also includes an I/O interface 216. I/O interface 216may be an interface for network I/O or disk I/O. For example, I/Ointerface 216 allows network data requests for data to be sent to (andresponses to be received from) other network nodes that arecommunicatively coupled to computing node 200. The network data requestsmay originate from worker thread 212. As another example, I/O interface216 allows storage I/O requests to be sent to (and responses to bereceived from) non-volatile storage (not depicted), which may beinternal or external to computing node 200. Again, storage I/O requestsmay originate from worker thread 212.

In an embodiment, I/O interface 216 corresponds to multiple separate I/Odevices. For example, different physical network devices may be combinedinto one logical I/O interface.

Computing node 200 also includes storage 218, which may be volatile ornon-volatile storage. Storage 218 stores results of requests (whethernetwork data requests or storage I/O requests). For example, resultsretrieved from another network node based on a particular network datarequest are stored in storage 218. The results may be stored inassociation with data that identifies the particular network datarequest. If worker thread 212 is currently processing a task that is“waiting” for results of an I/O request, then worker thread 212 checksstorage 218 for those results.

In an embodiment, computing node 200 includes a receiver thread (notdepicted) that is responsible for handling callbacks (such as I/Ocompletion events), from hardware of computing node 200, byacknowledging the callbacks. Doing so ensures that a queue for storingthe callbacks does not overrun.

In an embodiment, computing node 200 includes a communication thread(not depicted) that is responsible for receiving requests for data fromother network nodes, reads or writes the requested data in the localstorage, and response to the requests.

Example Process

FIG. 3 is a flow diagram that depicts a process 300 for reducingsynchronization of processing tasks, in an embodiment.

At block 310, multiple tasks are stored in task queue 214 to which onlyworker thread 212 is assigned for processing.

At block 320, worker thread 212 determines whether the task at the headof task queue 214 indicates an I/O operation, such as a network I/Ooperation or a disk I/O operation. If so, process 300 advances to block325. Else, process 300 advances to block 350. A task may indicate orrequire multiple I/O operations.

At block 325, it is determined whether the I/O operation has beenissued. If not, then process advances to block 330. Else, process 300advances to block 340.

At block 330, worker thread 212 generates an I/O request and causes theI/O request to be sent through I/O interface 216.

At block 340, worker thread 212 places the task at another position intask queue 214, such as the end of task queue 214, without placing thetask in any other queue. Block 340 may involve modifying one or morevariables of the task to indicate that the task is “waiting” for an I/Oresponse. If the task is not waiting for an I/O response and block 340is entered from block 360 (described below), then the task is modifiedto indicate what operation(s) of the task are left to perform and/orwhat operation(s) of the task have been performed. Whatever type ofoperation is performed for the task, a modification of the task mayinvolve updating state information that indicates which state (ofmultiple states of a state machine) the task is in.

Because I/O tasks are not inserted into another “waiting” queue that ismanaged by another thread, synchronization of such tasks is eliminatedentirely.

At block 350, worker thread 212 processes the task, which does notinvolve an IO operation. Instead, the task may require data that isstored in local (i.e., relative to network node) memory. Thus, workerthread 212 may access the memory and perform one or more operationsindicated by the task.

At block 360, worker thread 212 determines whether any more operationsneed to be performed for the task. If so, then process 300 proceeds toblock 340. Else, process 300 proceeds to block 370, where the task isremoved from task queue 214.

At block 380, worker thread determines whether any more tasks remain intask queue 214. This determination may be performed by comparing a headposition value (that indicates a position of the most recently-processedtask in task queue 214) with a tail position value (that indicates aposition of the last task in task queue 214). If, for example, the headposition value is not the same as the tail position value, then tasksremain in task queue 214. If the determination of block 380 is true,then process 300 proceeds to block 320 where the next task in task queue214 is analyzed.

Following process 300, the need for synchronization among multiplethreads is removed for both blocking (when a task is moved from a readyqueue to a wait queue) and wake-up (when the task is moved from the waitqueue to the ready queue). This allows task scheduling software to usesimpler and more efficient data structures. In addition, process 300removes the need for software to run upon I/O completions because thereis no need to perform a wake-up operation.

Intelligent Task Placement

As described previously, a task is reinserted into task queue 214 afterdetermining that the task is “waiting” for data from an I/O request.Instead of inserting the task at the end of task queue 214 (i.e., afterthe last task currently in task queue 214), the task may be insertedinto a “non-tail” position in task queue 214. For example, the task maybe inserted at a mid-point in task queue 214.

One or more factors may be taken into account to determine where toreinsert a task. Such factors may include I/O latency and task latency.I/O latency refers to an expected wait time before a response to an I/Orequest is available for an I/O task. I/O latency may change from timeto time based on current and/or previous statistics regarding actual I/Olatency. For example, initially, I/O latency may be 20 microseconds.Over time, however, I/O latency may increase to 95 microseconds.

Task latency refers to an expected time required to process each taskonce it reaches the head of task queue 214. Task latency may also changefrom time to time based on current task processing statistics.

For example, if there are 1,000 tasks in task queue 214, I/O latency is20 microseconds, and task latency is 90 nanoseconds, then it isestimated that 223 tasks in task queue 214 will be processed before thedata from an I/O request will be available for an I/O task that is atthe head of task queue 214. Therefore, the I/O task may be insertedimmediately after the 222^(nd) task in task queue 214. To account forunexpected delays or slowdowns in I/O traffic and/or for some tasksrequiring more complex operations than others, the I/O task may beinserted at a later position in task queue 214, such as 1.5× or 2× thecalculated position.

In an embodiment, the factor (e.g., whether 1.5× or 2×) is adapted inresponse to “early” or “late” execution of tasks. For example, workerthread 212 maintains a position estimate of 1.25× and increases theposition estimate if worker thread 212 determines that “waiting” I/Otasks are re-positioned in task queue 214. In other words, after beingrepositioned in task queue 214, I/O tasks are arriving at the head oftask queue 214 without their corresponding I/O operations beingcompleted yet. Alternatively, worker thread 212 decreases the positionestimate if worker thread 212 determines that “waiting” I/O tasks havecorresponding I/O operations that are always (or almost always, such as+90%) complete.

In order to allow insertion of a task at any arbitrary position, taskqueue 214 may be implemented as a linked list. However, the scatteredmemory consumption and traversal of linked lists may not be optimal.

If task queue 214 is implemented as an array (or vector), then, in orderto insert a task between two tasks, a set of tasks that will be afterthe reinserted task are copied from task queue 214 and reinserted (forexample, one position later) into task queue 214. Given the previousexample of 1,000 tasks, tasks at positions 223-1000 are copied and movedto positions 224-1001, respectively. The to-be-reinserted task is theninserted into the vacated position (i.e., 223 in this example).

Alternatively, instead of moving (potentially) many tasks for eachreinsertion, task queue 214 may be formed by leaving a certain number ofslots in task queue 214 available for reinsertions. For example, astasks are added to task queue 214, every tenth slot is left empty toallow a future I/O task to be inserted into one of the available slots.An available slot may be identified by first calculating a positionbased on I/O latency and task latency and then iteratively searching foran available slot that is after the calculated position. Alternatively,data that identifies available slots is maintained so that worker thread212 may quickly identify an available slot after calculating a positionbased on I/O latency and task latency. Alternatively, the task queue maybe represented by a linked list of array chunks, allowing tasks to beinserted between chunks in this list.

Regardless of how task queue 214 is implemented, the further a task isplaced from the front of task queue 214, the higher the time the taskcan be waiting in task queue 214 since the task's associated I/Ooperation has completed, but the lower the chance that worker thread 212will check for I/O results that are not yet available.

Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 4 is a block diagram that illustrates a computersystem 400 upon which an embodiment of the invention may be implemented.Computer system 400 includes a bus 402 or other communication mechanismfor communicating information, and a hardware processor 404 coupled withbus 402 for processing information. Hardware processor 404 may be, forexample, a general purpose microprocessor.

Computer system 400 also includes a main memory 406, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 402for storing information and instructions to be executed by processor404. Main memory 406 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 404. Such instructions, when stored innon-transitory storage media accessible to processor 404, rendercomputer system 400 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 400 further includes a read only memory (ROM) 408 orother static storage device coupled to bus 402 for storing staticinformation and instructions for processor 404. A storage device 410,such as a magnetic disk or optical disk, is provided and coupled to bus402 for storing information and instructions.

Computer system 400 may be coupled via bus 402 to a display 412, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 414, including alphanumeric and other keys, is coupledto bus 402 for communicating information and command selections toprocessor 404. Another type of user input device is cursor control 416,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 404 and forcontrolling cursor movement on display 412. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 400 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 400 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 400 in response to processor 404 executing one or more sequencesof one or more instructions contained in main memory 406. Suchinstructions may be read into main memory 406 from another storagemedium, such as storage device 410. Execution of the sequences ofinstructions contained in main memory 406 causes processor 404 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 410.Volatile media includes dynamic memory, such as main memory 406. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 402. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 404 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 400 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 402. Bus 402 carries the data tomain memory 406, from which processor 404 retrieves and executes theinstructions. The instructions received by main memory 406 mayoptionally be stored on storage device 410 either before or afterexecution by processor 404.

Computer system 400 also includes a communication interface 418 coupledto bus 402. Communication interface 418 provides a two-way datacommunication coupling to a network link 420 that is connected to alocal network 422. For example, communication interface 418 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 418 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 418sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 420 typically provides data communication through one ormore networks to other data devices. For example, network link 420 mayprovide a connection through local network 422 to a host computer 424 orto data equipment operated by an Internet Service Provider (ISP) 426.ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 428. Local network 422 and Internet 428 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 420and through communication interface 418, which carry the digital data toand from computer system 400, are example forms of transmission media.

Computer system 400 can send messages and receive data, includingprogram code, through the network(s), network link 420 and communicationinterface 418. In the Internet example, a server 430 might transmit arequested code for an application program through Internet 428, ISP 426,local network 422 and communication interface 418.

The received code may be executed by processor 404 as it is received,and/or stored in storage device 410, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A method comprising: storing a plurality of tasksin the queue of tasks; identifying, by a single thread, a particulartask that is at the head of the queue, wherein the particular task isone of the plurality of tasks; determining, by the single thread,whether the particular task requires an I/O operation; in response todetermining that the particular task requires an I/O operation:generating and sending an I/O request; without placing the particulartask in another queue, causing, by the single thread, the particulartask to be placed at a position, within the queue, that is not at thehead of the queue; wherein the method is performed by one or morecomputing devices.
 2. The method of claim 1, further comprising: aftercausing the particular task to be placed at the position, processing, bythe single thread, other tasks of the plurality of tasks withoutprocessing the particular task; wherein after processing the othertasks, the particular task is at the head of the queue; determining, bythe single thread, whether data associated with the I/O request isavailable.
 3. The method of claim 2, further comprising: in response todetermining that data associated with the I/O request is not available,causing, by the single thread, the particular task to be placed at asecond position, within the queue, that is not at the head of the queue.4. The method of claim 2, further comprising: in response to determiningthat data associated with the I/O request is available, processing, bythe single thread, the particular task based on the data associated withthe I/O request.
 5. The method of claim 4, further comprising: afterprocessing the particular task based on the data associated with the I/Orequest, determining, by the single thread, whether the particular taskrequires one or more other operations.
 6. The method of claim 1, whereincausing the particular task to be placed in the position within thequeue comprises causing, by the single thread, the particular task to beplaced at the end of the queue.
 7. The method of claim 1, whereincausing the particular task to be placed in the position within thequeue comprises causing, by the single thread, the particular task to beplaced at a position, within the queue, that is not the end of thequeue.
 8. The method of claim 7, further comprising: determining, by thesingle thread, the position based on one or more criteria; wherein theone or more criteria includes one or more of an estimated latency of anetwork, an estimated latency of each task, an estimated latency of astorage device, or a number of tasks in the queue.
 9. The method ofclaim 1, wherein the I/O operation is a network I/O operation or a diskI/O operation.
 10. The method of claim 1, further comprising: prior toidentifying the particular task, assigning the single thread to thequeue of tasks;
 11. The method of claim 1, further comprising: prior toidentifying the particular task, assigning, to the queue of task, aplurality of threads that are of the same processing core, wherein theplurality of threads includes the single thread.
 12. One or morenon-transitory computer-readable media carrying instructions which, whenexecuted by one or more processors, cause: storing a plurality of tasksin the queue of tasks; identifying, by a single thread, a particulartask that is at the head of the queue, wherein the particular task isone of the plurality of tasks; determining, by the single thread,whether the particular task requires an I/O operation; in response todetermining that the particular task requires an I/O operation:generating and sending an I/O request; without placing the particulartask in another queue, causing, by the single thread, the particulartask to be placed at a position, within the queue, that is not at thehead of the queue.
 13. The one or more non-transitory computer-readablemedia of claim 12, wherein the instructions, when executed by the one ormore processors, further cause: after causing the particular task to beplaced at the position, processing, by the single thread, other tasks ofthe plurality of tasks without processing the particular task; whereinafter processing the other tasks, the particular task is at the head ofthe queue; determining, by the single thread, whether data associatedwith the I/O request is available.
 14. The one or more non-transitorycomputer-readable media of claim 13, wherein the instructions, whenexecuted by the one or more processors, further cause: in response todetermining that data associated with the I/O request is not available,causing, by the single thread, the particular task to be placed at asecond position, within the queue, that is not at the head of the queue.15. The one or more non-transitory computer-readable media of claim 13,wherein the instructions, when executed by the one or more processors,further cause: in response to determining that data associated with theI/O request is available, processing, by the single thread, theparticular task based on the data associated with the I/O request. 16.The one or more non-transitory computer-readable media of claim 15,wherein the instructions, when executed by the one or more processors,further cause: after processing the particular task based on the dataassociated with the I/O request, determining, by the single thread,whether the particular task requires one or more other operations. 17.The one or more non-transitory computer-readable media of claim 12,wherein causing the particular task to be placed in the position withinthe queue comprises causing, by the single thread, the particular taskto be placed at the end of the queue.
 18. The one or more non-transitorycomputer-readable media of claim 12, wherein causing the particular taskto be placed in the position within the queue comprises causing, by thesingle thread, the particular task to be placed at a position, withinthe queue, that is not the end of the queue.
 19. The one or morenon-transitory computer-readable media of claim 18, wherein theinstructions, when executed by the one or more processors, furthercause: determining, by the single thread, the position based on one ormore criteria; wherein the one or more criteria includes one or more ofan estimated latency of a network, an estimated latency of each task, anestimated latency of a storage device, or a number of tasks in thequeue.
 20. The one or more non-transitory computer-readable media ofclaim 12, wherein the I/O operation is a network I/O operation or a diskI/O operation.
 21. The one or more non-transitory computer-readablemedia of claim 12, wherein the instructions, when executed by the one ormore processors, further cause: prior to identifying the particulartask, assigning the single thread to the queue of tasks;
 22. The one ormore non-transitory computer-readable media of claim 12, wherein theinstructions, when executed by the one or more processors, furthercause: prior to identifying the particular task, assigning, to the queueof task, a plurality of threads that are of the same processing core,wherein the plurality of threads includes the single thread.
 23. Acomputer system comprising: one or more processors; one or morecomputer-readable media storing instructions which, when executed by theone or more processors, cause: storing a plurality of tasks in the queueof tasks; identifying, by a single thread, a particular task that is atthe head of the queue, wherein the particular task is one of theplurality of tasks; determining, by the single thread, whether theparticular task requires an I/O operation; in response to determiningthat the particular task requires an I/O operation: generating andsending an I/O request; without placing the particular task in anotherqueue, causing, by the single thread, the particular task to be placedat a position, within the queue, that is not at the head of the queue.